MrHen comments on Welcome to Heaven - Less Wrong

23 Post author: denisbider 25 January 2010 11:22PM

You are viewing a comment permalink. View the original post to see all comments and the full post content.

Article Navigation

Comments (242)

Comments (242)

Sort By: Best

Show:

You are viewing a single comment's thread. Show more comments above.

Comment author: denisbider 26 January 2010 04:05:21PM * -1 points [-]

Well...

Consider you want to explore and learn and build ad infinitum. Progress in your activities requires you to control increasing amounts of matter and consume increasing amounts of energy, until such point as you conflict with others who also want to build and explore. When that point is reached, the only way the FAI can make you all happy is to intervene while you all sleep, put you in separate vats, and from then on let each of you explore an instance of the universe that it simulates for you.

Should it let you wage Star Wars on each other instead? And how would that be different from no AI to begin with?

Comment author: RobinZ 26 January 2010 04:24:42PM 2 points [-]

You seem to be engaging in all-or-nothing thinking. Because I want more X does not mean that I want to maximize X to the exclusion of all other possibilities. I want to explore and learn and build, but I also want to act fairly toward my fellow sapients/sentients. And I want to be happy, and I want my happiness to stem causally from exploring, learning, building, and fairness. And I want a thousand other things I'm not aware of.

An AI which examines my field of desires and maximizes one to the exclusion of all others is actively inimical to my current desires, and to all extrapolations of my current desires I can see.

Comment author: denisbider comment score below threshold [+] (9 children)

Comment author: RobinZ 26 January 2010 05:12:40PM * 0 points [-]

If you seek personal growth, then the AI already is everything you can aspire to be.

Why would I build an AI which would steal everything I want to do and leave me with nothing worth doing? That doesn't sound like the kind of future I want to build.

Edit:

But everything you do is temporary. All the results you get from it are temporary.

That just adds a constraint to what I may accomplish - it doesn't change my preferences.

Comment author: denisbider 26 January 2010 05:22:38PM -1 points [-]

Why would I build an AI which would steal everything I want to do and leave me with nothing worth doing? That doesn't sound like the kind of future I want to build.

Because only one creature can be maximized, and it's better it's an AI than a person.

Even if we don't necessarily want the AI to maximize itself immediately, it will always need to be more powerful than any possible threat, and therefore more powerful than any other creature.

If you want the ultimate protector, it has to be the ultimate thing.

Comment author: RobinZ 26 January 2010 05:32:53PM 3 points [-]

I don't want it maximized, I want it satisficed - and I, at least, am willing to exchange a small existential risk for a better world. "They who can give up essential liberty to obtain a little temporary safety" &c.

If the AI can search the universe and determine that it is adequately secure from existential threats, I don't want it expanding very quickly beyond that. Leave some room for us!

Comment author: denisbider 26 January 2010 05:39:04PM * 0 points [-]

But the AI has to plan for a maximized outcome until the end of the universe. In order to maximize the benefit from energy before thermal death, resource efficiency right now is as important as when resources will be scarcest.

This is unless the AI discovers that thermal death can be overcome, in which case, great! But what we know so far indicates that the universe will eventually die, even if many billions of year in the future. So conservative resource management is important from day 1.

Comment author: RobinZ 26 January 2010 05:48:34PM 0 points [-]

There are things I could say in reply, but I suspect we are simply talking past each other. I may reply later if I have some new insight into the nature of our disagreement.

Comment author: denisbider 26 January 2010 05:51:38PM * -1 points [-]

The way I understand our disagreement is, you see FAI as a limited-functionality add-on that makes a few aspects of our lives easier for us, while I see it as an unstoppable force, with great implications for everything in its causal future, which just can't not revolutionize everything, including how we feel, how we think, what we do. I believe I'm following the chain of reasoning to the end, whereas you appear to think we can stop after the first couple steps.

Comment author: Vladimir_Nesov 27 January 2010 04:17:28PM * 5 points [-]

You also keep asserting to know in which particular way FAI is going to change things. Instead of repeating the same statements, you should recognise the disagreement, and address it directly, instead of continuing to profess the original assertions.

Comment author: AdeleneDawner 26 January 2010 06:10:03PM 3 points [-]

One quality that's relevant to Friendly AI is that it does stop, when appropriate. It's entirely plausible (according to Eliezer; last time I checked) that a FAI would never do anything that wasn't a response to an existential threat (i.e. something that could wipe out or severely alter humanity), if that was the course of action most in keeping with our CEV.

Comment author: RobinZ 26 January 2010 08:12:33PM * 2 points [-]

I don't think that's the source of our disagreement - as I mentioned in another thread, if prudence demanded that the population (or some large fraction thereof) be uploaded in software to free up the material substance for other purposes, I would not object. I could even accept major changes to social norms (such as legalization of nonconsensual sex, to use Eliezer Yudkowsky's example). Our confirmed point of disagreement is not your thesis that "a human population which acquired an FAI would become immensely different from today's", it is your thesis that "a human population which acquired an FAI would become wireheads". Super Happy People, maybe - not wireheads.

MrHen comments on Welcome to Heaven - Less Wrong

Article Navigation

Comments (242)

Nearest Meetups

Virtual Study Room

Recent Comments

Recent Posts

Recent Rationality Quotes

Recent Wiki Edits

Top Contributors, 30 Days

Recent Karma Awards

Comments (242)

Top Contributors, 30 Days

You'll need to login or register to do that

(Don't worry, it only takes a few seconds)

Create

Login